AI Lessons
Lesson 18: Detecting objects in real time

Purpose: To use AI to detect and track objects in real time through a webcam feed and build interactive applications that combine visual bounding boxes with voice announcements.

No. of Classes

1 - (Time : 1 hour 30 minutes, Laptops/desktops : 10, Students strength : 15 to 20).

Materials Required

Laptop / Desktop with Internet connection / Wi-Fi.

webcam and speakers.

Curriculum Content

4. Working with Pre-trained AI Models

  • 4.1 Pre-trained models and inference
  • 4.2 Real-time input from camera and microphone
  • 4.5 Object detection and localisation
  • 4.8 Mapping model output to an interface
  • 4.10 Speech synthesis

7. Designing AI-enabled Systems

  • 7.3 Integrating models into applications
Prior knowledge
  • Bounding box concepts (x, y, w, h) from previous lesson
  • Basic canvas drawing in p5.js (`rect`, `text`, `stroke`)
  • Handling webcam permissions in the browser
Exercises

Exercise (1) - Live Bounding Box Overlay


  • Click here to download the base code for exercise-1
  • Open the base code in VS Code and initialize the ml5.js object detection model (e.g., COCO-SSD) on the webcam video stream.
  • Write a loop to iterate through the returned array of detected objects.
  • For every detected object, draw a colored rectangle around its boundary using its x, y, width, and height coordinates.
  • Display the detected object's label name and confidence score percentage above each box.

Exercise (2) - Audio Announcements with Text-to-Speech


  • Integrate the Web Speech API (`window.speechSynthesis`) into your p5.js sketch.
  • Configure the application to speak the label of a newly detected object when it first enters the frame (e.g., "Person detected", "Cell phone detected").
  • Add state logic so the computer speaks the announcement only once when an object arrives, rather than repeating it constantly on every frame.

Solutions


Teacher's Instruction:
  1. Demonstrate ml5.js Real-Time Object Detection (Exercise 1):
    • Show students how the model continuously outputs an array of object results on every webcam frame.
    • Walk through accessing object properties in code: `results[i].label`, `results[i].x`, `results[i].y`, `results[i].width`, and `results[i].height`.
    • Remind students to match the canvas size to the webcam video feed size so the drawn boxes overlap perfectly with the real objects.
  2. Guide Text-to-Speech Integration & Debouncing (Exercise 2):
    • Explain how `speechSynthesis.speak()` works in browser JavaScript.
    • Highlight a common bug: if `speak()` is called inside the continuous `draw()` loop without a check, the voice will overlap uncontrollably.
    • Show students how to use a simple tracking variable or array (e.g., `lastSpokenLabel`) to ensure the system only speaks when a new object appears or changes.
  3. Explore Edge Cases in Real-Time Video:
    • Have students move objects quickly across the camera view and observe frame rate lag or bounding box "ghosting."
    • Discuss why live video requires the computer to make predictions 15 to 30 times per second compared to analyzing a static image once.
  4. Points to Ponder:
    • How could combining object detection with voice announcements help visually impaired individuals navigate an unfamiliar room? What mistakes must the system avoid to be safe?